{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用朴素贝叶斯算法对中文文档进行分类\n",
    "\n",
    "   朴素贝叶斯分类最适合的场景就是文本分类、情感分析和垃圾邮件识别。这也是朴素贝叶斯最擅长的地方。所以朴素贝叶斯也是常常用于自然语言处理的工具。\n",
    "   \n",
    "**常见的贝叶斯算法：**\n",
    "\n",
    "高斯朴素贝叶斯： 特征变量是连续的变量，符合高斯分布，比如说人的身高，物体的长度。\n",
    "\n",
    "伯努利朴素贝叶斯： 特征变量是布尔变量，符合0/1分布，在文档分类中特征是单词是否出现。\n",
    "\n",
    "多项式朴素贝叶斯： 特征变量是离散型的变量，符合多项式分布，在文档分类中特征变量体现在一个单词出现的次数，或者是单词的TF-IDF值等。（本例采用）\n",
    "\n",
    "\n",
    "\n",
    "## TF-IDF值（词频TF和逆向文档频率IDF乘积）\n",
    "\n",
    "   TF-IDF用来评估某个词语对于一个文件集或者文档库中的其中一份文件的重要程度。有助于选取词汇作为特征。\n",
    "   TF-IDF实际上是两个词组Term Frequency和Inverse Document Frequency的总称，两者缩写为TF和IDF，分别代表了词频和逆向文档频率。\n",
    "\n",
    "**词频TF**计算了一个单词在文档中的出现的次数，它认为一个单词的重要性和它在文档中出现的次数成正比。\n",
    "\n",
    "**逆向文档频率IDF：**是指一个单词在文档中的区分度。它认为一个单词出现在的文档数越少，就越能通过这个单词把该文档和其他文档区分开。IDF越大就代表该单词的区分度越大。\n",
    "\n",
    "所以**TF-IDF是词频TF和逆向文档频率IDF的乘积。**我们倾向于找到TF和IDF取值都很高的单词作为区分，即这个单词在一个文档中出现的次数多，同时又和少出现在其他文档中。这样的单词适合用于分类。\n",
    "\n",
    "\n",
    "\n",
    "## TF-IDF 如何计算\n",
    "\n",
    "首先我们看下词频TF和逆向文档概率IDF的公式\n",
    "$$ 词频TF =  \\frac{单词出现的次数}{该文档的总单词数}$$\n",
    "$$ 逆向文档频率IDF = log\\frac{文档总数}{该单词出现的文档数+1}$$\n",
    "为什么IDF的分母中，单词出现的文档数加1？因为有些单词可能不会存在文档中，为了避免分母为零，统一给单词出现的文档数都加1\n",
    "\n",
    "TF-IDF = TF*IDF\n",
    "我们可以看到，TF—IDF的值就是TF与IDF的成绩，这样可以更加准确的对文档进行分类。比如“我”这样的高频单词，虽然TF词频高，但是IDF值很低，整体的TF-IDF也不高。\n",
    "\n",
    "**例如** ：假设一个文件夹里面一共有10篇文档，其中一篇文档有1000个单词，“this”这个单词出现20次，“bayes”出现5次。“this”在所有的文档中均出现过，而“bayes”只在两篇文档中出现。我们可以计算一下这两个词语的TF-IDF的值。\n",
    "\n",
    "针对“this”，计算TF-IDF值：\n",
    "$$词频TF=\\frac{20}{1000} = 0.02$$\n",
    "$$逆向文档的频率IDF=log\\frac{10}{10+1} = -0.0414$$\n",
    "\n",
    "所以TF-IDF = 0.02*(-0.0414) = -8.28e-4\n",
    "\n",
    "针对“bayes”，计算TF-IDF值：\n",
    "针对“bayes”，计算TF-IDF值：\n",
    "\n",
    "$$词频TF=\\frac{5}{1000} = 0.005$$\n",
    "$$逆向文档的频率IDF=log\\frac{10}{2+1} = 0.5229$$\n",
    "\n",
    "\n",
    "TF-IDF=0.005*0.5229=2.61e-3\n",
    "\n",
    "很明显“bayes”的TF-IDF的值要大于“this”的TF-IDF值。这就说明用“bayes”这个单词做区分比单词“this”要好。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TfidfVectorizer 类的创建：\n",
    "\n",
    "\n",
    "有两个构造参数，可以自定义停用词stop_words和规律规则token_pattern。需要注意的是传递的数据结构，停用词stop_words是一个列表List类型，而过滤规则token_pattern是正则表达式。\n",
    "\n",
    "停用词就是在分类中没有用的词，这些词一般词频TF高，但是IDF很低，起不到分类的作用。为了节省空间和计算时间，告诉机器这些词不需要帮我计算。\n",
    "\n",
    "|参数表|作用|\n",
    "|--|--|\n",
    "|stop_words|自定义停用词表，为列表list类型|\n",
    "|token_pattern|过滤规则，正则表达式，如r\"(?u)\\b\\w+\\b\"|\n",
    "\n",
    "当我们创建好TF-IDF向量类型时候，可以使用fit_transform帮助我们计算，返回给我们文本矩阵，该矩阵表示了每一个单词在每个文档中的TF-IDF值。\n",
    "\n",
    "|方法表|作用|\n",
    "|--|--|\n",
    "|fit_transform(X)|拟合模型，并返回文本矩阵|\n",
    "\n",
    "\n",
    "在我们进行fit_transform()拟合模型之后，我们可以得到更多的TF-IDF向量属性，比如，我们可以得到词汇的对应关系（字典类型）和向量类型的IDF，当然也可以获取设置的停用词stop_words.\n",
    "\n",
    "|属性表|作用|\n",
    "|--|--|\n",
    "|vocabulary_|词汇表；字典型|\n",
    "|idf_|返回idf值|\n",
    "|stop_words_|返回停用词表|\n",
    "\n",
    "\n",
    "1、**基于分词的数据准备**，包括分词、单词的权重计算、去掉停用词。\n",
    "\n",
    "2、**应用朴素贝叶斯分类进行分类**，首先通过训练集得到贝叶斯分类器，然后将分类器应用于测试集，并与实际结果做对比，最终得到测试集的分类准确率。\n",
    "\n",
    "### 采用jieba中文分词库\n",
    "\n",
    "\n",
    "## 主要模块\n",
    "\n",
    "### 模块1：中文文档分词加载停用词\n",
    "\n",
    "1.在中文文档中，最常用的就是jieba包。jieba包中包含了中文的停用词stop words和分词方法。\n",
    "2.读取停用词表文件，从网上可以找到中文常用的停用词保存在stop_words.txt,然后利用Python的文件读取函数读取文件，保存在stop_words数组中。\n",
    "\n",
    "### 模块2：计算单词的权重，获取特征\n",
    "\n",
    "直接创建TfidfVectorizer类，然后使用fit_transform方法进行拟合，得到TF-IDF特征空间features，你可以理解为选出来的分词就是特征。我们计算这些特征在文档上面的特征向量，得到特征空间features。\n",
    "\n",
    "### 模块3：生成朴素贝叶斯分类器\n",
    "\n",
    "采用的是多项式贝叶斯分类器，其中alpha为平滑参数。为什么需要平滑呢？因为如果一个单词在训练样本中没有出现，这个单词的概率就会被计算为0。但是训练集样本只是整体的抽样情况，我们不能因为一个事件没有被观察到，就认为整个事件的概率为0.为了解决整个问题，我们需要做平滑处理。\n",
    "\n",
    "当alpha=1时候，使用的是Laplace平滑。Laplace平滑就是采用加1的方式，来统计没有出现过的单词的概率。这样当训练的样本很大的时候，加1得到的概率变化可以忽略不计，也同时避免了0概率的问题。\n",
    "\n",
    "当0<alpha<1时候，使用的是Lidstone平滑。对于Lidstone平滑来说，alpha越小，迭代的次数越多，精度就越高。我们可以设置alpha为0.001.\n",
    "\n",
    "### 模块4：分类器预测\n",
    "\n",
    "首先我们需要得到测试集的特征矩阵test_features.\n",
    "\n",
    "方法是用训练集的分词创建一个TfidfVectorizer类，使用同样的stop_words和max_df,然后用这个TfidfVectorizer类对测试集的内容进行fit_transform拟合，得到测试集的特征矩阵test_features.\n",
    "\n",
    "### 模块5：分类器评价\n",
    "\n",
    "accuracy_score函数评分，混淆矩阵和分类样本评估报告classification_report。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 中文文本分类\n",
    "# coding: utf-8\n",
    "import os\n",
    "import jieba\n",
    "import warnings\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "def cut_words(file_path):\n",
    "    \"\"\"\n",
    "    对文本进行切词\n",
    "    :param file_path:txt 文本路径\n",
    "    :return:使用空格分词的字符串\n",
    "    \"\"\"\n",
    "    text_with_spaces = ''\n",
    "    # 这里会遇到字符集编码问题，所以直接采用二进制方式读取\n",
    "    text = open(file_path, 'rb').read()\n",
    "    textcut = jieba.cut(text)\n",
    "    for word in textcut:\n",
    "        text_with_spaces += word + ' '\n",
    "    return text_with_spaces\n",
    "\n",
    "\n",
    "def loadfile(file_dir, label):\n",
    "    \"\"\"\n",
    "    将路径下的所有文件加载\n",
    "    :param file_dir: 保存txt文件目录\n",
    "    :param label: 文档标签\n",
    "    :return: 分词后的文档列表和标签\n",
    "    \"\"\"\n",
    "    file_list = os.listdir(file_dir)\n",
    "    words_list = []\n",
    "    labels_list = []\n",
    "    for file in file_list:\n",
    "        file_path = file_dir + '/' + file\n",
    "        words_list.append(cut_words(file_path))\n",
    "        labels_list.append(label)\n",
    "    return words_list, labels_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache C:\\Users\\SCOTTL~1\\AppData\\Local\\Temp\\jieba.cache\n",
      "Loading model cost 0.640 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    }
   ],
   "source": [
    "# 训练数据\n",
    "train_words_list1, train_labels1 = loadfile('./text_classification/train/女性', '女性')\n",
    "train_words_list2, train_labels2 = loadfile('./text_classification/train/体育', '体育')\n",
    "train_words_list3, train_labels3 = loadfile('./text_classification/train/文学', '文学')\n",
    "train_words_list4, train_labels4 = loadfile('./text_classification/train/校园', '校园')\n",
    "\n",
    "train_words_list = train_words_list1 + train_words_list2 + train_words_list3 + train_words_list4\n",
    "train_labels = train_labels1 + train_labels2 + train_labels3 + train_labels4\n",
    "\n",
    "# 测试数据\n",
    "test_words_list1, test_labels1 = loadfile('./text_classification/test/女性', '女性')\n",
    "test_words_list2, test_labels2 = loadfile('./text_classification/test/体育', '体育')\n",
    "test_words_list3, test_labels3 = loadfile('./text_classification/test/文学', '文学')\n",
    "test_words_list4, test_labels4 = loadfile('./text_classification/test/校园', '校园')\n",
    "\n",
    "test_words_list = test_words_list1 + test_words_list2 + test_words_list3 + test_words_list4\n",
    "test_labels = test_labels1 + test_labels2 + test_labels3 + test_labels4\n",
    "\n",
    "stop_words = open('./text_classification/stop/stopword.txt', 'r', encoding='utf-8').read()\n",
    "stop_words = stop_words.encode('utf-8').decode('utf-8-sig')  # 列表头部\\ufeff处理\n",
    "stop_words = stop_words.split('\\n')  # 根据分隔符分隔\n",
    "\n",
    "# 计算单词权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准确率为： 0.91\n",
      "\n",
      "混淆矩阵：\n",
      " [[106   3   5   1]\n",
      " [  1  36   1   0]\n",
      " [  1   0  30   0]\n",
      " [  3   1   2  10]]\n",
      "\n",
      "分类报告:\n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "          体育       0.95      0.92      0.94       115\n",
      "          女性       0.90      0.95      0.92        38\n",
      "          文学       0.79      0.97      0.87        31\n",
      "          校园       0.91      0.62      0.74        16\n",
      "\n",
      "    accuracy                           0.91       200\n",
      "   macro avg       0.89      0.87      0.87       200\n",
      "weighted avg       0.92      0.91      0.91       200\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf = TfidfVectorizer(stop_words=stop_words,  max_df=0.5)\n",
    "\n",
    "train_features = tf.fit_transform(train_words_list)\n",
    "# 上面fit过了，所以这里直接transform,下面的是测试的\n",
    "test_features = tf.transform(test_words_list)\n",
    "\n",
    "# 多项式贝叶斯分类器\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB(alpha=0.001).fit(train_features, train_labels)\n",
    "predicted_labels = clf.predict(test_features)\n",
    "\n",
    "# 计算准确率\n",
    "print('准确率为：', metrics.accuracy_score(test_labels, predicted_labels))\n",
    "print('\\n混淆矩阵：\\n',confusion_matrix(test_labels, predicted_labels))\n",
    "print('\\n分类报告:\\n',metrics.classification_report(test_labels, predicted_labels))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}